Repository navigation
API: Raise when mixing SLEEF and longdouble backends - #122
SwayamInSync wants to merge 22 commits into
Conversation
…ing through double
…ly in the resolver
|
I believe the earlier behaviour was pretty non-deterministic and buggy, while I will be working on adding support of proper intermediate (f64) free inter-backend working (planned for future releases). We can strictly error-out the any inter-backend use for now instead of being sloppy with the implementation |
|
Sorry for taking a little while to look at this! I agree that fixing this is worthwhile. Unfortunately the current approach is overly broad and breaks valid operations like this: a = np.array([1, 2], dtype=QuadPrecDType("longdouble"))
a + 1 # Raises “Cannot mix QuadPrecision backends”I think adding this restriction needs to wait for a NumPy fix. NumPy materializes Another problem is this: x = QuadPrecision(1, backend="longdouble")
QuadPrecision(x) # Defaults to SLEEF and raisesIf a backend is already selected, the constructor should use that instead of the default. The prototype below defers conversion of the original Python scalar until the input descriptor is resolved, avoiding the intermediate SLEEF conversion. NumPy already uses this ordering in its private scalar-aware resolver path. An explicit DType opt-in preserves existing user dtypes’ behavior. With quaddtype opted in, literals work with either backend, while actual mixed-backend arrays still raise. The NumPy patch: opt-in conversion of Python scalars using resolved descriptorsdiff --git a/numpy/_core/include/numpy/dtype_api.h b/numpy/_core/include/numpy/dtype_api.h
index 047cb861b9..89bf9c2059 100644
--- a/numpy/_core/include/numpy/dtype_api.h
+++ b/numpy/_core/include/numpy/dtype_api.h
@@ -411,6 +411,9 @@ typedef int (PyArrayMethod_PromoterFunction)(PyObject *ufunc,
#define NPY_DT_PARAMETRIC 1 << 2
#define NPY_DT_NUMERIC 1 << 3
+/* Opt in to packing Python numeric scalars after descriptor resolution. */
+#define NPY_DT_REPACK_PYSCALARS 1 << 4
+
/*
* These correspond to slots in the NPY_DType_Slots struct and must
* be in the same order as the members of that struct. If new slots
diff --git a/numpy/_core/src/multiarray/multiarraymodule.c b/numpy/_core/src/multiarray/multiarraymodule.c
index 516986d2d5..b17d029c42 100644
--- a/numpy/_core/src/multiarray/multiarraymodule.c
+++ b/numpy/_core/src/multiarray/multiarraymodule.c
@@ -2033,6 +2033,10 @@ array_copyto(PyObject *NPY_UNUSED(ignored),
if (descr == NULL) {
goto fail;
}
+ if (is_pyscalar && (dst_DType->flags & NPY_DT_REPACK_PYSCALARS) &&
+ NPY_DTYPE(descr) == dst_DType) {
+ Py_SETREF(descr, (PyArray_Descr *)Py_NewRef(PyArray_DESCR(dst)));
+ }
int res = npy_update_operand_for_scalar(&src, src_obj, descr, casting);
Py_DECREF(descr);
if (res < 0) {
diff --git a/numpy/_core/src/multiarray/public_dtype_api.c b/numpy/_core/src/multiarray/public_dtype_api.c
index ebf0440fc7..f2c5633eab 100644
--- a/numpy/_core/src/multiarray/public_dtype_api.c
+++ b/numpy/_core/src/multiarray/public_dtype_api.c
@@ -57,11 +57,13 @@ PyArrayInitDTypeMeta_FromSpec(
}
/* Check and handle flags: */
- int allowed_flags = NPY_DT_PARAMETRIC | NPY_DT_ABSTRACT | NPY_DT_NUMERIC;
+ int allowed_flags = NPY_DT_PARAMETRIC | NPY_DT_ABSTRACT | NPY_DT_NUMERIC
+ | NPY_DT_REPACK_PYSCALARS;
if (spec->flags & ~(allowed_flags)) {
PyErr_SetString(PyExc_RuntimeError,
"invalid DType flags specified, only NPY_DT_PARAMETRIC, "
- "NPY_DT_ABSTRACT, and NPY_DT_NUMERIC are valid flags for "
+ "NPY_DT_ABSTRACT, NPY_DT_NUMERIC, and NPY_DT_REPACK_PYSCALARS "
+ "are valid flags for "
"user DTypes.");
return -1;
}
diff --git a/numpy/_core/src/umath/ufunc_object.c b/numpy/_core/src/umath/ufunc_object.c
index 67ad4da489..f1e797cc5a 100644
--- a/numpy/_core/src/umath/ufunc_object.c
+++ b/numpy/_core/src/umath/ufunc_object.c
@@ -4456,6 +4456,7 @@ resolve_descriptors(int nop,
/* number of original_descrs entries to clean up, including NULLs */
int n_cleanup = 0;
PyArray_Descr *original_descrs[NPY_MAXARGS];
+ npy_bool deferred_scalars[NPY_MAXARGS] = {0};
NPY_UF_DBG_PRINT("Resolving the descriptors\n");
@@ -4544,6 +4545,15 @@ resolve_descriptors(int nop,
if (new_descr == NULL) {
goto finish;
}
+ if (input != NULL &&
+ (PyArray_FLAGS(operands[i]) & NPY_ARRAY_WAS_PYTHON_LITERAL) &&
+ NPY_DTYPE(new_descr) == signature[i] &&
+ (signature[i]->flags & NPY_DT_REPACK_PYSCALARS)) {
+ original_descrs[i] = new_descr;
+ n_cleanup = i + 1;
+ deferred_scalars[i] = NPY_TRUE;
+ continue;
+ }
int res = npy_update_operand_for_scalar(
&operands[i], input, new_descr, casting);
Py_DECREF(new_descr);
@@ -4601,6 +4611,14 @@ resolve_descriptors(int nop,
ufunc_get_name_cstr(ufunc), npy_casting_to_string(casting));
goto finish;
}
+ /* Pack deferred Python scalars once the input descriptors are resolved. */
+ for (int i = 0; i < ufunc->nin; i++) {
+ if (deferred_scalars[i] && npy_update_operand_for_scalar(
+ &operands[i], inputs[i], dtypes[i], casting) < 0) {
+ goto finish;
+ }
+ }
+
retval = 0;
finish:
diff --git a/numpy/_core/tests/test_custom_dtypes.py b/numpy/_core/tests/test_custom_dtypes.py
index e6c5d0583c..7e27113961 100644
--- a/numpy/_core/tests/test_custom_dtypes.py
+++ b/numpy/_core/tests/test_custom_dtypes.py
@@ -198,6 +198,12 @@ def test_sfloat_cast_internal_errors(self, aligned):
match="error raised inside the core-loop: non-finite factor!"):
a.astype(SF(2e-300))
+ def test_copyto_python_scalar_keeps_existing_cast_path(self):
+ # Without opt-in, scalar assignment must still execute the cast.
+ target = np.zeros(1, dtype=SF(1e-310))
+ with pytest.raises(TypeError, match="non-finite factor"):
+ np.copyto(target, 1e-310)
+
def test_sfloat_promotion(self):
assert np.result_type(SF(2.), SF(3.)) == SF(3.)
assert np.result_type(SF(3.), SF(2.)) == SF(3.)Quaddtype opts in by adding We should delay the blanket rejection until this support is available in the NumPy versions quaddtype supports. Representation-corruption fixes can proceed separately. The constructor regression affecting scalar-returning reductions also needs its own fix. |
|
I’ll take your |
Addresses #113 for now, until #112 makes casts between backends exact.
Values of the SLEEF and longdouble backends currently move between each other in two flawed ways. Every conversion goes through
float64(#112), so it silently loses precision in both directions. Operations also disagree on the common backend (#113):np.result_typeandnp.concatenatepick SLEEF, while ufuncs pick longdouble.This PR makes the policy consistent by refusing every such operation with a
TypeErroruntil there is an exact conversion and one agreed promotion rule. Everything that does not combine the two backends is unchanged, and NumPy's own dtypes, includingnp.longdouble, work with both backends as before.Behaviour
Upstream
main(888042c) vs this branch, on x86-64 Linux:mainnp.result_type(sleef_dt, ld_dt)QuadPrecDType(backend='sleef')TypeError(sleef_arr + ld_arr).dtypeQuadPrecDType(backend='longdouble')TypeErrornp.concatenate([sleef_arr, ld_arr]).dtypeQuadPrecDType(backend='sleef')TypeError1 + 2**-112.astype(ld_dt), minus 10.0TypeError1 + 2**-60.astype(sleef_dt), minus 10.0TypeErrorld_arr[0] = QuadPrecision(1.5)0.0TypeErrornp.sqrt(sleef [4.0], out=ld_arr, casting="unsafe")0.0TypeErrorQuadPrecision(q) == qforq = 1 + 2**-112FalseTrueld_arr + 1[2.5]TypeErrorld_arr + np.float64(1)[2.5][2.5]ld_arr.astype(np.longdouble)[1.5][1.5]The error is
TypeError: Cannot mix QuadPrecision backends 'sleef' and 'longdouble'. Where NumPy checks the casting rule before calling our resolver (gufuncs,astype(..., casting="same_kind")), it is NumPy's ownCannot cast ...TypeErrorinstead.Backwards-incompatible changes
out=arrays have different backends,astype,np.copyto, slice and item assignment,np.result_type,np.promote_types,np.concatenate, andQuadPrecision(value, backend=...)given aQuadPrecisionof the other backend now raise.np.arrayof mixedQuadPrecisionscalars gives an object array.intandfloatuse the default SLEEF descriptor, so they raise with longdouble arrays too. This includes NumPy functions that pass Python scalars internally:np.ones,np.full,np.ones_like,mean,var,std,np.median,np.isclose,np.allcloseandnp.round. The workarounds are NumPy scalars or arrays, e.g.ld / np.float64(2),np.full(3, np.longdouble(1), dtype=ld.dtype)ornp.ones(3).astype(ld.dtype). SLEEF arrays are unaffected.Bug fixes
QuadPrecisioninto an array of the other backend, or casting an object array of them, stored the bytes under the wrong backend (0.0instead of1.5above).out=array of the other backend andcasting="unsafe"wrote values in the wrong format.QuadPrecision(q)rounded a same-backendqthroughfloat64; it now copies exactly.Implementation
quad_to_quad_resolve_descriptorsis the single point that refuses a cross-backend cast. The cast loop's cross-backend branch andquad_to_quad_same_value_checkare kept as they are, so BUG: inter-backend casts lose precision by converting through double #112 only has to replace thefloat64conversions and allow the cast in the resolver.common_instance, the binary, comparison and unary resolvers,setitemand the scalar paths raise through shared helpers indtype.h(quad_set_backend_mismatch_error,quad_loop_backend,quad_out_loop_descr).float64-based same_value tests for cross-backend casts are kept as strict xfails taggedgh-112. Once casts between backends are allowed, they XPASS and fail CI until the markers are removed.Tests and docs
tests/test_backend_mixing.pycovers every path above, the literal limitation, the workarounds, and that NumPy dtypes are unaffected. The new tests fail onmain.docs/api/core.md, and the changes above in the README changelog under "Unreleased".CI
big_endian.ymlskips pandas 3.1.0 on s390x. Its sdist does not build there (pandas-dev/pandas#71548, fix planned for 3.1.1), and s390x has no pandas wheels.pandas!=3.1.0keepstest_pandas_strreprunning on 3.0.6 and moves to 3.1.1 automatically.AI Disclosure
Used claude code as assistant for testing and writing PR/docs